Ensemble Methods Based on Bias–variance Analysis Title: Ensemble Methods Based on Bias–variance Analysis

Ensembles of classifiers represent one of the main research directions in machine learning. Two main theories are invoked to explain the success of ensemble methods. The first one consider the ensembles in the framework of large margin classifiers, showing that ensembles enlarge the margins, enhancing the generalization capabilities of learning algorithms. The second is based on the classical bias–variance decomposition of the error, and it shows that ensembles can reduce variance and/or bias. In accordance with this second approach, this thesis pursues a twofold purpose: on the one hand it explores the possibility of using bias–variance decomposition of the error as an analytical tool to study the properties of learning algorithms; on the other hand it explores the possibility of designing ensemble methods based on bias–variance analysis of the error. At first, bias–variance decomposition of the error is considered as a tool to analyze learning algorithms. This work shows how to apply Domingos and James theories on bias–variance decomposition of the error to the analysis of learning algorithms. Extended experiments with Support Vector Machines (SVMs) are presented, and the analysis of the relationships between bias, variance, kernel type and its parameters provides a characterization of the error decomposition, offering insights into the way SVMs learn. In a similar way bias–variance analysis is applied as a tool to explain the properties of ensembles of learners. A bias–variance analysis of ensembles based on resampling techniques is conducted, showing that, as expected, bagging is a variance reduction ensemble method, while the theoretical property of canceled variance holds only for Breiman’s random aggregated predictors. In addition to analyzing learning algorithms, bias–variance analysis can offer guidance to the design of ensemble methods. This work shows that it provides a theoretical and practical tool to develop new ensemble methods well-tuned to the characteristics of a specific base learner. On the basis of the analysis and experiments performed on SVMs and bagged ensembles of SVMs, new ensemble methods based on bias–variance analysis are proposed. In particular Lobag (Low bias bagging ) selects low bias base learners and then combines them through bootstrap aggregating techniques. This approach affects both bias, through the selection of low bias base learners, and variance, through bootstrap aggregation of the selected low bias base learners. Moreover a new potential class of ensemble methods (heterogeneous ensembles of SVMs), that aggregate different SVM models on the basis of their bias– variance characteristics, is introduced. From an applicative standpoint it is also shown that the proposed ensemble methods can be successfully applied to the analysis of DNA microarray data.